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Abstract 


The  paper  describes  reflectance  analysis  of  real  images  taken  in  an  outdoor  environment. 
This  is  one  of  the  first  attempts  for  analyzing  reflection  in  outdoor  images.  In  the  past,  the 
fundamental  difficulties  involved  in  analyzing  outdoor  images  have  prevented  the  use  of  tra¬ 
ditional  techniques  for  reflectance  analysis  such  as  shape  from  shading  and  photometric  ste¬ 
reo.  Those  difficulties  include:  1.  multiple  reflection  components  due  to  two  light  sources  of 
different  spectral  and  spatial  distributions,  namely  the  sunlight  and  the  skylight,  2.  ambigu¬ 
ity  in  surface  normal  determination  caused  by  the  sun’s  restricted  motion  on  the  ecliptic,  and 
3.  non-uniform  surface  albedo  of  object  surfaces.  In  order  to  overcome  those  difficulties,  we 
developed  a  reflectance  model  under  solar  illumination.  Based  on  this  model,  we  proposed 
solutions  for  each  of  these  problems.  We  have  successfully  tested  our  algorithms  using  real 
color  images  taken  both  in  a  laboratory  setup  simulating  sunlight,  and  in  an  outdoor  envi¬ 
ronment  under  solar  illumination. 
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1  Introduction 

Techniques  to  recover  object  shapes  from  reflectance  and  shading  information,  contained  in  either 
a  single  intensity  image  or  multiple  intensity  images,  are  categorized  into  two  main  groups.  One  is 
called  shape  from  shading,  and  the  other  is  generally  known  as  photometric  stereo.  Horn  origi¬ 
nally  introduced  the  shape  from  shading  technique  [4]  which  recovered  object  shape  from  a  single 
intensity  image  by  using  the  characteristic  strip  expansion  method.  Following  Horn’s  method, 
many  other  methodologies  have  been  proposed  by  other  researchers.  (For  instance,  the  chapter  of 
shape  from  shading  in  [2]  is  a  good  source  of  information  about  the  shape  from  shading  tech¬ 
niques.)  Woodham  [15]  proposed  a  new  technique  called  photometric  stereo  for  shape  recovery. 
This  technique  uses  multiple  images  with  different  light  source  locations.  Later,  Nayar,  Ikeuchi 
and  Kanade  [10]  developed  a  technique  for  recovering  object  shape  and  reflectance  without  any 
knowledge  of  surface  reflectance. 

These  techniques  for  shape  recovery  have  been  successfully  applied  to  intensity  images  taken  in  a 
laboratory  setup.  However,  reports  of  applications  for  real  intensity  images  of  outside  scenes  have 
been  very  limited.  Intensity  images  synthesized  or  taken  in  a  laboratory  setup  are  well  controlled 
and  are  less  complex  than  those  taken  outside  under  sunlight.  For  instance,  in  an  outdoor  environ¬ 
ment,  there  are  multiple  light  sources  of  different  colors  and  spatial  distributions,  namely  the  sun¬ 
light  and  the  skylight.  The  sunlight  can  be  regarded  as  yellow  point  light  source  whose  movement 
is  restricted  to  the  ecliptic1.  On  the  other  hand,  the  skylight  is  a  blue  extended  light  source  which 
is  almost  constant  over  the  entire  hemisphere.  Due  to  the  sun’s  restricted  movement,  the  problem 
of  surface  normal  recovery  becomes  underconstrained  under  sunlight.  For  instance,  if  the  photo¬ 
metric  stereo  method  is  applied  to  two  intensity  images  taken  outside  at  different  times,  two  sur¬ 
face  normals  which  are  symmetric  with  respect  to  the  ecliptic  are  obtained  at  each  surface  point. 
Those  two  surface  normals  cannot  be  distinguished  locally  because  those  two  surface  normal 
directions  give  us  exactly  the  same  brightness  at  the  surface  point. 

Another  factor  that  makes  reflectance  analysis  under  the  solar  illumination  different  is  multiple 
reflection  components  from  the  object  surface.  Reflection  from  object  surfaces  may  have  multiple 


1.  The  great  circle  of  the  celestial  sphere  that  is  the  apparent  path  of  the  sun  among  the  stars  or  of  the  earth  as  seen  from 
the  sun:  the  plane  of  the  earth’s  orbit  extended  to  meet  the  celestial  sphere. 
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reflection  components  such  as  the  diffuse  reflection  component  and  the  specular  reflection  com¬ 
ponent.  The  two  reflection  components  are  predicted  from  the  dichromatic  reflectance  model 
which  was  proposed  by  Shafer  [13].  Recently,  Sato  and  Ikeuchi  [12]  proposed  and  successfully 
demonstrated  a  methodology  to  separate  the  two  reflection  components  at  each  pixel.  They  used  a 
color  image  sequence  taken  with  a  moving  light  source.  The  main  advantage  of  the  technique  is 
that  it  does  not  require  any  a-priori  knowledge  of  surface  reflectance,  and  only  a  sequence  of 
observed  color  vectors  at  each  pixel  is  necessary. 

In  other  situations,  the  object  may  be  painted  in  different  colors.  In  other  words,  the  object  has  a 
non-uniform  surface  albedo.  In  this  case,  the  traditional  techniques  for  shape  recovery  cannot  be 
correctly  applied  because  those  methods  are  based  on  the  assumption  of  a  uniform  surface  albedo. 
To  compensate  for  this  effect,  the  surface  albedo  has  to  be  normalized  before  shape  recovery.  If 
the  surface  albedo  varies  gradually  on  the  object  surface,  the  normalization  can  be  quite  difficult. 
However,  if  the  problem  is  limited  to  special  cases  where  the  object  surface  consists  of  patches 
with  constant  surface  albedo,  the  non-uniform  surface  albedo  can  be  normalized  by  using  several 
techniques.  Horn  [3]  used  Land’s  retinex  theory  [6]  to  determine  surface  albedo  which  is  insensi¬ 
tive  to  an  illumination  change.  By  using  this  technique,  the  surface  albedo  can  be  obtained  for 
each  uniform  albedo  region.  Then,  the  surface  albedo  can  be  normalized. 

In  this  paper,  we  address  the  three  issues  involved  in  analyzing  real  outdoor  intensity  images 
taken  under  the  solar  illumination.  1.  the  multiple  reflection  components  including  highlight  (the 
specular  reflection  component  according  to  the  dichromatic  reflection  model.)  2.  the  normaliza¬ 
tion  of  surface  albedo.  3.  the  unique  solution  for  surface  normals  under  sunlight.  For  each  of  these 
problems,  we  show  a  solution  and  demonstrate  the  feasibility  of  the  solution  by  using  images 
which  are  taken  in  a  laboratory  setup  and  outdoors  under  the  sun. 

The  paper  is  organized  as  follows.  The  section  2  describes  the  reflectance  model  that  we  used  for 
analyzing  outdoor  images  under  solar  illumination.  The  development  of  the  model  is  based  on  the 
dichromatic  reflection  model  and  takes  into  account  two  light  sources  of  different  spectral  and 
spatial  distributions.  In  the  section  3,  a  method  to  remove  the  specular  reflection  component  by 
using  a  sequence  of  color  images  is  explained.  The  normalization  of  the  surface  albedo  is 
described  in  the  section  4.  A  method  to  obtain  two  sets  of  surface  normals  for  the  object  surface  to 
choose  the  correct  set  of  surface  normals  are  discussed  in  the  section  5.  Experimental  results  from 
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a  laboratory  setup  and  the  outdoor  environment  will  be  shown  in  the  section  6  and  the  section  7, 
respectively.  Concluding  remarks  are  presented  in  the  section  8. 
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2  Reflection  model  under  solar  illumination 

2.1  Reflection  model  with  a  single  light  source 

In  this  section,  a  reflectance  model  with  a  single  light  source  is  described.  We  will  develop  a 
reflectance  model  under  solar  illumination  in  the  section  2.2. 

A  mechanism  of  reflection  is  described  in  terms  of  three  reflection  components,  namely  the  spec¬ 
ular  spike,  the  specular  lobe,  and  the  diffuse  lobe.  These  reflection  components  can  be  represented 
by  the  Beckmann-Spizzichino  model  [1],  the  Torrance-Sparrow  model  [14]  (or  the  Beckmann- 
Spizzino  model),  and  the  Lambertian  model,  respectively  [10].  The  model  of  reflection  which 
accounts  for  all  three  components  tends  to  be  too  complicated  to  be  used  as  a  model  for  actual 
applications.  Thus,  a  simplified  model  which  represents  two  reflection  components,  the  specular 
reflection  component  and  the  diffuse  reflection  component,  is  used  in  our  analysis.  The  specular 
reflection  component  of  the  simplified  reflection  model  represents  a  combination  of  the  specular 
spike  and  the  specular  lobe.  The  combined  specular  reflection  component  is  modeled  as  a  simple 
function  such  as  an  impulse  function.  Nayar  called  the  reflectance  model  the  hybrid  reflectance 
model  [10].  In  this  reflectance  model,  the  pixel  intensity  of  an  image  is  expressed  as 

I  =  ID  +  IS  (EQ1> 

where  IQ  is  the  diffuse  reflection  component  and  Is  is  the  specular  reflection  component. 

In  order  to  simplify  the  discussion,  we  develop  the  reflectance  model  in  a  two  dimensional  planer 
case  instead  of  the  three  dimensional  case.  An  equivalent  discussion  can  be  applied  to  the  three 
dimensional  case  without  fundamental  differences. 

The  dichromatic  reflection  model  in  the  two  dimensional  planer  case  can  be  represented  by  the 
bidirectional  spectral-reflectance  distribution  function  (BSRDF)  as: 

fr  (0/,  er>  X)  =  cD  (to  g  D  (0/>  er)  +  (X)  gs(  ef,  er)  (EQ2) 

where  fr  is  the  BSRDF,  c£)(A,)and  c5(X)are  the  spectral  reflectance  distributions, 
gD  0P  and  Ss  (6,->  Op  are  the  geometrical  terms.  The  subscripts  D  and  S  refer  to  the  diffuse 
reflection  component  and  the  specular  reflection  component,  respectively.  0  and  0  are  the  inci- 
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dent  angle  and  the  reflecting  angle  as  shown  in  Figure  1 . 


Figure  1  Reflectance  model  with  two  reflection  components 

In  most  reflectance  models,  it  is  assumed  that  the  diffuse  reflection  component  has  a  spectral  dis¬ 
tribution  different  from  that  of  an  incident  light,  while  the  specular  reflection  component  has  a 
similar  spectral  distribution.  Lee  called  this  model  the  neutral-interface-reflection  (NIR)  model 
[8].  Considering  the  NIR  model,  (EQ2)  is  written  as: 

fr  (0 .,  0r,  X)  =  cD  {X)  gD  (0 .,  0r)  +  %  (0 .,  0r)  (EQ3) 

Since  we  assumed  that  the  diffuse  reflection  component  is  modeled  by  the  Lambertian  model,  and 
the  specular  reflection  component  is  modeled  by  a  unit  impulse  function,  (EQ3)  becomes 

fr(0p  0r,  X)  =  cD(X)cos0.  +  c55(0.-0r)  (EQ4) 

On  the  other  hand,  the  intensity  of  incident  light  onto  the  object  surface  (and  not  into  the  camera) 
is  represented  as: 

L.(0.,X)  =  c(X)L(0.)  (EQ5) 

where  c  (X)  is  the  spectral  distribution  of  the  incident  light  and  L  (0 .)  is  a  geometrical  term  of 
incident  light  onto  the  object  surface.  The  intensity  of  light  reflected  by  the  object  surface  and 
coming  into  the  camera  can  be  expressed  as  the  product  of  Lf  (0 .,  X)  and  the  BSRDF 
/  (0 0r,  X)  .  Finally,  the  pixel  intensity  I  is  given  by: 
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7(0;,  0r)  =  Jj  (A)  L.  (0.,  A)/r  (0.,  dr,  A)  dl 

X 

=  (A)  c  (A)  L  (0f.)  {cDcos0.  +  c58(0(.-0r)  }  d\ 

X 

=  L  (0.)  cos0.Js  (A)  c  (A)  cD{'k)d'k  +  L  (0.)  8  (0.  -  0r)  c  (A)  c  (A)  dX 


(EQ6) 


where  s  (A)  is  the  spectral  response  of  a  camera. 


2.2  Reflectance  model  with  two  light  sources:  the  sun  and  the  sky 

The  reflectance  model  with  a  single  light  source  was  described  in  the  previous  section.  In  this  sec¬ 
tion,  we  extend  the  model  for  the  case  of  two  light  sources  of  different  spectral  and  spatial  distri¬ 
butions.  In  our  analysis,  these  two  light  sources  are  the  sunlight  and  the  blue  skylight. 

One  significant  feature  of  the  skylight  is  its  uniformity  over  the  entire  hemisphere.  The  skylight  is 
highly  scattered  and  it  appears  to  be  almost  constant.  For  this  reason,  the  spatial  distribution  of  the 
skylight  Lsky  appears  to  be  constant.  As  a  result,  the  intensity  of  incident  light  (EQ5)  in  the  case 
of  the  sunlight  and  the  skylight  is  represented  as: 

Li  (9/’  =  Csun  ^  Lsurt  <0/)  +  Csky  M  Lsky  (EQ7> 


On  the  other  hand,  the  sunlight  is  almost  a  point  light  source.  However,  it  has  a  finite  size,  so  we 
use  a  narrow  Gaussian  distribution  (Figure  2)  to  model  the  sunlight  spatial  distribution.  The  cen¬ 
ter  of  the  distribution  is  the  sun’s  direction  0  . 


L,««(0,-0P  =AexP 


f  -(0.-0  )2^ 
v  i  s' 

2  ) 

n  / 


(EQ8) 


By  incorporating  this  light  model  (EQ7)  into  (EQ6),  the  pixel  intensity  is  given  by: 
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/(es,0r)  =  p„„  ( 0,  -  et)  COS0  Yisj?  a>  <:S[IH  (A)  cD  (A)  dX 
+ (e,  -  es)  5  (0i  -  er)  ae  J,  «  csun  W  di 

+  Lskyj cos0i dSilS  csky^  CD  (l)  dk 

tVsl8(e.-9r)4(l>%(l,A  (EQ9) 

Unfortunately,  jLsun  (0.-  0?)  cos 6^8 f  in  the  first  term  of  the  right  side  of  (EQ9)  is  difficult  to 
evaluate  analytically.  However,  we  found  that  a  cosine  curve  could  be  a  good  numerical  approxi¬ 
mation  of  the  integral  as  shown  in  Figure  3.  Other  terms  are  simplified  further.  We  finally  get  the 
pixel  intensity  as: 

i'(0s,er)  =  KsunDcosds  +  KsmSLsm(6r-es)  +Ksky  (EQ10) 


where 

KsimD  ~  B jS  W  Csun  (X)  CD  (X)  dX 
KsunS  = 

Ksky  =  Xsky{lsWcst/X)cDa)dX  +  ls(X)csma)dl) 


(EQ11) 


It  is  important  to  see  that  the  two  reflection  components  from  the  skylight  is  constant  with  respect 
to  the  direction  of  the  sun  0^  and  the  viewing  direction  0r-  B  is  a  scaling  factor  due  to  the 
approximation  of  the  geometry  term  of  the  sunlight  diffuse  reflection  by  a  cosine  function.  Chang¬ 
ing  the  coordinate  system  into  the  viewer-centered  coordinate  system  illustrated  in  Figure  4, 
(EQ11)  becomes 

'  (8.)  =  KsunDcos  (0S  -  0„)  +  KsunSLsm  (20„  -  0,)  +  Ksky  (EQ12) 

In  our  analysis,  the  reflectance  model  represented  as  (EQ12)  is  used  to  remove  the  specular  reflec¬ 
tion  component  and  for  the  shape  recovery.  Separation  of  the  specular  reflection  component  and 
the  diffuse  reflection  component  will  be  discussed  in  the  section  3.  Shape  recovery  under  sunlight 
will  be  explained  in  the  section  5. 
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L 

sun 


Figure  2  The  distribution  of  the  sunlight  (0?  =  0,  a  =  0.01  n) 


K„(e,-e,)coSe,rf9 

A  cos  0^. 


Figure  3  The  diffuse  reflection  of  the  sun  and  a  cosine  curve 


camera 


light  source 


Figure  4  The  viewer-centered  coordinate  system 
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3  The  specular  reflection  component  removal  from  a 
sequence  of  color  images 

The  algorithm  to  remove  the  specular  reflection  component  from  the  sunlight  is  described  in  this 
section.  The  input  to  the  algorithm  is  a  sequence  of  outdoor  color  images  taken  at  different  times 
(i.e.  every  15  minutes)  on  the  same  day. 

3.1  Estimation  of  the  reflection  component  from  the  skylight 

As  stated  in  the  section  2.2,  the  diffuse  and  specular  reflection  components  from  skylight  are  con¬ 
stant  with  respect  to  the  sun  direction  0  and  the  surface  normal  direction  0^ .  Therefore,  shadow 
regions  from  the  sunlight  have  uniform  pixel  intensities  since  they  are  illuminated  only  by  the 
skylight.  Pixel  intensities  in  those  regions  don’t  have  the  reflection  components  from  the  sunlight. 
They  only  have  the  reflection  components  from  the  skylight  K.  .  For  this  reason,  the  value  of 
the  reflection  component  due  to  the  skylight  Ksk  can  be  obtained  as  an  average  pixel  intensity  in 
the  shadow  regions  of  constant  pixel  intensity  as  shown  in  Color  Figure  6.  Ksk  is  subtracted 
from  all  pixel  intensities  of  the  image  to  yield 

1  (8.)  =  KsunDcos  (8, -  8„)  +  KsunSLsun  (28„  -  8,)  (EQ13) 

Then,  the  pixel  intensity  has  only  the  diffuse  and  specular  reflection  components  from  sunlight. 

3.2  Separation  of  the  two  reflection  component  from  the  sunlight 

The  algorithm  to  separate  the  two  reflection  components  from  the  sunlight  is  described  here.  The 
algorithm  was  originally  introduced  by  Sato  and  Ikeuchi  in  [12]. 

Using  red,  green,  and  blue  filters,  the  coefficients  KsunD  and  KsunS,  in  (EQ13),  become  two  lin¬ 
early  independent  vectors,  K  D  and  KsunS,  unless  the  colors  of  the  two  reflection  components 
are  accidentally  the  same: 
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\TR{X)s(X)csun{X)cDa)dX 

k 

X 

sunDR 

K sunD 

k 

K sunDG 

= 

Jt  G{'k)s{'k)csun{\)cD{X)dX 

k 

KsunDB 

X 

j^R  (^)  S  (^)  C sun  (^)  C D  (^)  ^ 

X 


CJT/?(^)5(^)  Csun 

k 

X 

sunSR 

K sunS 

k sunSG 

= 

CsjXG  5  csun  ^  ^ 

sun  SB 

X 

csjxB(l)s(X)csuna)dX 

X 


(EQ14) 


(EQ15) 


These  two  vectors  represent  the  colors  of  the  diffuse  and  specular  reflection  components  in  the 
dichromatic  reflectance  model  [13]. 

First,  the  pixel  intensities  in  the  R,  G,  and  B  channels  with  m  different  light  source  directions,  are 
measured  at  one  pixel.  It  is  important  to  note  that  all  intensities  are  measured  at  the  same  pixel. 
The  three  sequences  of  intensity  values  are  stored  in  the  columns  of  an  m  x  3  matrix  M .  Consid¬ 
ering  the  hybrid  reflectance  model  and  two  color  vectors  in  (EQ13),  (EQ14),  and  (EQ15),  the 
intensity  values  in  the  R,  G,  and  B  channels  can  be  represented  as: 


M  - 


Mr  Mq  Mr 


COS  (0  ,  -0  )  L  (0  .  -20  ) 

v  .si  rr  sun  v  si  n ’ 

COS  (0  0  -  0  )  L  (6  -  -  20  ) 

v  s 2  n '  sun  v  s2  n ’ 

COS  (0  -0  )  L  (0  -20  ) 

v  sm  n'  sunK  sm  n' 


k  k  k 

sunDR  sunDG  KsunDB 

k  k  k 

sunSR  * sunSG  KsunSB 
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where  the  two  vectors  GsunD  and  G  s  represent  the  intensity  values  of  the  diffuse  and  specular 
reflection  components  with  respect  to  the  direction  of  the  sun  0  ? .  Vector  KsunD  represents  the 
diffuse  reflection  color  vector.  Vector  K  s  represents  the  specular  reflection  color  vector.  We 
call  the  two  matrices  G  and  K,  the  geometry  matrix  and  the  color  matrix,  respectively. 

Suppose  we  have  an  estimation  of  the  color  matrix  K .  Then,  the  two  reflection  components  repre¬ 
sented  by  the  geometry  matrix  G  are  obtained  by  projecting  the  observed  reflection  stored  in  M 
onto  the  two  color  vectors  KsunD  and  KsunS  • 

G  =  MK+  (EQ17) 

where  K+  is  a  3  x  2  pseudoinverse  matrix  of  the  color  matrix  K. 

The  derivation  shown  above  is  based  on  the  assumption  that  the  color  matrix  K  is  known.  The 
method  to  estimate  the  color  matrix  K,  more  specifically  the  two  color  vectors  ^sunD  and 
K  is  discussed  in  the  following  sections. 

~  sunS 

3.2.1  Estimation  of  the  illuminant  color 

Several  algorithms  have  been  proposed  to  estimate  illuminant  color.  The  method  used  by  Lee  [7] 
is  briefly  described  here.  According  to  the  dichromatic  reflectance  model  [13],  the  color  of  reflec¬ 
tion  from  a  dielectric  object  is  a  linear  combination  of  the  diffuse  reflection  component  and  the 
specular  reflection  component.  The  color  of  the  specular  reflection  component  is  equal  to  the  illu¬ 
minant  color  (EQ15).  In  the  x-y  chromaticity  diagram,  the  observed  color  of  the  dielectric  object 
lies  on  a  segment  whose  endpoints  represent  the  colors  of  the  diffuse  and  specular  reflection  com¬ 
ponents.  By  representing  the  color  of  each  object  as  a  segment  in  the  chromaticity  diagram,  the 
illuminant  color  can  then  be  determined  from  the  intersection  of  the  two  segments  attributed  to 
the  two  objects  of  interest  [6]. 

In  our  experiment,  the  row  K sun ^  of  the  color  matrix,  K ,  which  represents  illumination  color,  is 
estimated  by  using  a  method  similar  to  the  method  described  above.  First,  several  pixels  of  differ¬ 
ent  colors  in  the  image  are  manually  selected  (Color  Figure  1).  The  observed  reflection  color  from 
those  selected  pixels  is  a  linear  combination  of  the  diffuse  reflection  component  and  the  specular 
reflection  component.  By  plotting  the  observed  reflection  color  of  each  pixel  in  the  x-y  chromatic¬ 
ity  diagram  over  the  image  sequence,  we  obtain  several  line  segments  in  the  x-y  chromaticity  dia- 
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gram.  The  illuminant  color  can  then  be  determined  by  the  intersection  of  those  line  segments  in 
the  diagram.  This  is  shown  in  Figure  6  for  the  case  of  the  real  image  shown  in  Color  Figure  1. 

This  technique  is  limited  to  the  case  that  there  are  objects  of  different  colors  in  the  image.  In  other 
words,  if  the  image  contains  objects  of  only  one  color,  the  light  source  color  cannot  be  estimated. 
In  those  cases,  the  illumination  color  is  obtained  by  measuring  the  color  vector  of  the  light  source 
directly.  In  our  experiment  for  outdoor  color  images  which  will  be  shown  in  the  section  7,  the  illu¬ 
mination  color  of  the  sunlight  is  directly  measured  by  using  a  standard  color  chart. 


3.2.2  Estimation  of  the  diffuse  reflection  color 


The  other  row  KsunD  of  the  color  matrix  cannot  be  obtained  in  the  same  manner  because  it 
depends  on  the  material  of  the  object.  Fortunately,  in  order  to  solve  this  problem,  we  can  use  the 
fact  that  the  distribution  of  the  specular  reflection  component  for  the  extended  light  source  is  lim¬ 
ited  to  a  fixed  angle,  depending  on  a  in  (EQ8).  Therefore,  if  two  vectors 
r  -171 

w 


i  l !Ri  JGi  1  Bi 


(i  -  1,  2)  are  sampled  on  the  0^,  axis  at  large  enough  interval,  at  least  one 


of  these  vectors  will  be  equal  to  the  color  vector  of  the  diffuse  reflection  component  .  This 


vector  has  no  specular  reflection  component.  The  desired  color  vector  of  the  diffuse  reflection 
T 

component  £J(|nD  is  the  vector  w.  which  subtends  the  largest  angle  with  respect  to  the  vector 
T 

K sunS .  The  angle  between  the  two  color  vectors  can  be  calculated  as: 


P  =  acos 


T 

K 

~ sunS  ~i 


T 

-sunS 

|u\| 

(EQ18) 


Once  we  get  the  color  matrix  K,  the  geometry  matrix  G  can  be  calculated  from  (EQ17). 


M diffuse  ~ 

(EQ19) 

^ surface  ~ 

(EQ20) 

After  the  reflection  components  from  the  skylight  and  the  specular  reflection  component  from  the 
sunlight  are  removed  by  the  method  explained  in  the  section  3,  the  pixel  intensities  in  the  image 
are  modeled  by  the  equation: 


page  13 
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^ sunDR 

w 

= 

K sunDG 

C0S(^-en) 

(EQ21) 

W 

V 

sunDB 

Since  the  pixel  intensity  now  has  only  the  diffuse  reflection  component  from  the  sunlight,  the 
intensities  in  three  color  bands  are  redundant  for  the  purpose  of  shape  recovery.  Thus,  only  one 
band  of  the  three  color  bands  is  used  in  our  discussions  on  surface  albedo  normalization  and  sur¬ 
face  normal  determination. 

'<»,)  =  *„„Dcos(es-e„) 


(EQ22) 
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4  The  normalization  of  surface  albedo  k  n 

suriD 


In  this  section,  the  method  to  normalize  the  surface  albedo  K '  D  is  discussed.  The  shape  recov¬ 
ery  algorithm  will  be  explained  in  the  section  5.  The  object  image  with  non-uniform  surface 
albedo  (Color  Figure  2)  is  normalized  to  yield  the  image  of  the  object  with  uniform  surface  albedo 
(Figure  8). 

The  object  in  the  image  may  not  have  a  uniform  surface  albedo,  i.e.  several  regions  of  different 
colors  may  be  on  the  object  surface.  If  this  is  the  case,  the  shape  recovery  method  based  on  the 
assumption  of  a  uniform  surface  albedo  will  fail.  Therefore,  we  need  to  apply  a  method  to  normal¬ 
ize  the  surface  albedo  to  obtain  an  image  for  shape  recovery  as  if  the  object  had  a  surface  with  a 
uniform  surface  albedo  K  „ . 

sunD 

Recently,  Nayar  and  Bolle  [9]  proposed  to  use  a  ratio  of  surface  albedos  of  neighboring  uniform 
surface  albedo  regions  for  object  recognition  purposes.  A  similar  technique  is  used  for  our  analy¬ 
sis,  in  this  case,  the  normalization  of  the  surface  albedo.  First,  the  object  surface  is  segmented  into 
regions  with  uniform  albedos  divided  by  a  boundary  where  a  pixel  intensity  changes  abruptly. 
Consider  two  neighboring  pixels  at  the  boundary.  One  pixel  lies  in  the  region  A  and  the  other 
exists  in  the  region  B .  The  two  points  on  the  object  surface  which  correspond  to  the  two  image 
pixels  are  assumed  to  have  the  same  surface  normal,  for  a  smooth  continuous  surface.  In  this  case, 
the  intensities  of  the  two  pixels  are  given  from  (EQ22)  as: 


/  (0p  =  l^sunDCOS  (0^  -  6^  (EQ23) 

I8  (0,)  =  K8 sunDCOS  (Qs  -  0^  (EQ24) 


As  you  can  see  in  the  equations  above,  the  ratio  of  the  two  surface  albedos  K^unD  and  K^unD  is 
equal  to  the  ratio  of  the  two  pixel  intensities  and  ^ .  This  is  because  the  two  surface  points 
corresponding  to  the  two  image  pixels  have  the  same  surface  normal  direction  0  . 


C  / 

K B 


sunD 


lB 


(EQ25) 
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This  surface  albedo  ratio  is  computed  at  all  points  along  the  boundary  between  the  region  A  and 
the  region  B .  Then,  the  surface  albedo  ratio  of  the  two  regions  is  given  as  the  average  of  all  com¬ 
puted  ratios.  By  repeating  the  same  procedure  for  all  pair  of  two  neighboring  regions  in  the  image, 
we  obtain  a  table  of  surface  albedo  ratios.  The  surface  albedo  ratios  are  propagated  in  the  table,  so 
that  surface  albedo  ratio  of  all  pairs  of  any  two  regions  are  obtained  unless  some  regions  are  com¬ 
pletely  isolated  from  others.  For  instance,  consider  the  two  separated  regions  A  and  B,  and  the 
region  C  between  the  two  regions.  The  surface  reflectance  ratio  between  the  region  A  and  B  is 
computed  as: 


1  sunD 
^sunD 


if  n  KC  n 

sunD  sunD 

KC  '  J3 

A sunD  sunD 


(EQ26) 


The  surface  albedo  ratio  can  be  computed  by  using  another  technique.  The  use  of  the  Land’s  ret- 
inex  theory  [6]  for  calculating  surface  albedo  was  first  introduced  by  Horn  [3].  In  this  algorithm, 
the  Laplacian  operator  is  first  applied  to  the  logarithm  of  image  pixel  intensities  in  order  to 
emphasize  edges.  The  result  is  thresholded  to  remove  smooth  intensity  fluctuations  due  to  spatial 
variations  in  the  incident  light  distribution.  Finally,  the  logarithm  of  lightness  is  estimated  by  con¬ 
volving  the  result  with  the  inverse  Laplacian  operator.  By  using  the  algorithm,  Horn  successfully 
obtained  the  image  intensity  which  is  an  estimation  of  the  surface  albedo.  The  estimation  of  sur¬ 
face  albedo  can  be  used  for  normalizing  the  surface  albedo  prior  to  shape  recovery  in  our  analysis 
of  outdoor  scene.  Unfortunately,  we  found  that  the  algorithm  based  on  the  Land’s  retinex  theory  is 
not  as  robust  as  the  other  method  described  in  this  section  for  most  images.  Therefore,  we  decided 
to  use  the  method  to  measure  the  surface  albedo  ratio  directly. 
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5  Obtaining  surface  normals 

5.1  Two  sets  of  surface  normals 

After  the  specular  reflection  removal  and  the  surface  albedo  normalization,  the  input  image 
sequence  has  only  the  diffuse  reflection  component  from  the  sunlight.  Usually,  shape  from  shad¬ 
ing  and  photometric  stereo  are  used  for  recovering  shape  information  from  diffuse  reflection 
images.  Initially,  those  techniques  are  implemented  for  shape  recovery  in  our  experiments.  How¬ 
ever,  we  unfortunately  found  that  neither  of  those  techniques  could  yield  correct  object  shapes.  It 
is  attributed  to  various  sources  of  noise  in  the  input  image  such  as  incomplete  removal  of  the 
specular  reflection  component.  Shape  from  shading  and  photometric  stereo  use  very  small  number 
of  images  for  surface  normal  computation.  That  leads  us  to  an  erroneous  object  shape  when  the 
images  contain  slight  errors  in  pixel  intensities.  Therefore,  we  decided  to  use  another  algorithm  to 
determine  surface  normals  from  the  input  image  sequence.  The  algorithm  makes  use  of  more 
images  in  the  sequence,  rather  than  just  few  of  them.  We  describe  the  algorithm  in  this  section. 


v: viewing  direction 


Figure  5  Sun  direction,  viewing  direction  and  surface  normal  in  3D  case 

To  represent  the  sun’s  motion  in  three  dimensional  space,  we  consider  the  Gaussian  sphere  as 
shown  in  Figure  5.  The  ecliptic  is  represented  as  the  great  circle  on  the  Gaussian  sphere  The  view¬ 
ing  direction  v  is  fixed.  The  direction  of  the  sun  s  is  specified  as  the  function  of  6  in  the  plane  of 
the  ecliptic. 
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Consider  an  intensity  of  one  pixel  as  the  function  of  the  sun  direction  7(0^)  .  If  the  maxi¬ 
mum  intensity  is  observed  when  the  sun  is  located  at  the  direction  Qg' ,  the  surface  normal  of 
the  image  pixel  should  be  located  somewhere  on  the  great  circle  P{P2  which  is  perpendicu¬ 
lar  to  the  ecliptic.  For  obtaining  robust  estimations,  the  maximum  pixel  intensity  I'  and  the 
direction  of  the  sun  0  '  are  found  by  fitting  a  second  degree  polynomial  to  the  observed 

pixel  intensity  sequence.  According  to  the  reflectance  model  (EQ22),  the  angle  between  the 

— >  —■ > 

sun  direction  s  and  the  surface  normal  directions  n{  and  n2  on  the  great  circle  P \P2  1S 
given  by: 

a  =  acos  f  (EQ27) 

^ KsunD ' 

The  surface  albedo  K  unD  has  to  be  known  for  computing  a .  If  we  assume  that  at  least  one 
surface  normal  on  the  object  surface  is  the  same  as  the  sun  direction  s ,  the  surface  albedo 
K  D  is  simply  obtained  as  the  intensity  of  the  pixel  I  =  KgunD .  The  pixel  in  the  image 
can  be  found  simply  as  the  brightest  pixel.  In  a  practical  case,  the  estimation  of  the  surface 
albedo  is  computed  as  the  average  of  the  brightest  pixel  intensities  from  multiple  images  of 
the  input  image  sequence,  for  robustness.  We  empirically  found  the  algorithm  described  in 
this  section  works  better  for  estimating  surface  normals  in  our  analysis. 

5.2  Unique  surface  normal  solution 

Due  to  the  sun’s  restricted  movement  on  the  ecliptic,  we  cannot  obtain  a  unique  solution  for 
surface  normal  by  applying  photometric  stereo  to  outdoor  images  taken  at  different  times  at 
the  same  day.  This  fact  was  pointed  out  by  Woodham  [15]  when  he  introduced  the  photo¬ 
metric  stereo  method.  As  a  result,  there  has  been  no  attempts  reported  for  recovering  an 
object  shape  by  the  photometric  stereo  method  applied  to  outdoor  images.  However,  Onn 
and  Bruckstein  [11]  recently  studied  photometric  stereo  applied  to  two  images  and  showed 
that  surface  normals  can  be  determined  uniquely  even  if  only  two  images  are  used,  except 
some  special  cases. 

— > 

By  using  the  algorithm  described  in  the  previous  section,  two  sets  of  surface  normals  nl 
and  ~n2  are  obtained.  We  used  the  constraint  which  Onn  called  integrability  constraint,  in 
order  to  choose  a  correct  set  of  surface  normals  out  of  the  two  sets  of  surface  normals. 
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The  Onn’s  integrability  constraint  is  described  briefly  here.  First,  we  compute  two  surface  nor- 
— >  — > 

mals  n{  and  «2  for  all  pixels.  Then,  the  object  surface  is  segmented  into  subregions  by  defining  a 
boundary  where  two  surface  normals  are  similar.  In  practice,  if  an  angle  between  and  ~n2  is 
within  a  threshold,  the  pixel  is  included  in  the  boundary.  Then,  for  each  subregion  R ,  two  inte¬ 
grals  are  computed. 
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(at,  j)  e  R 


dpl  dq{ 
dy  dx  j 


dxdy 
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(EQ29) 


(EQ30) 


Theoretically,  the  correct  set  of  surface  normals  produces  the  integral  value  equal  to  zero.  In  a 
practical  case,  the  correct  surface  normal  set  can  be  chosen  as  the  one  with  the  integral  value  close 
to  zero.  Onn  and  Bruckstein  showed  that  the  integrability  constraint  is  always  valid  except  for  a 
few  rare  cases  where  the  object  surface  can  be  represented  as  H  (x,  y)  =  F  (x)  +  G  (y)  in  a  suit¬ 
ably  defined  coordinate  system.  In  our  experiments,  the  exceptional  case  does  not  occur,  so  the 
integrability  constraint  can  be  used  for  obtaining  a  unique  solution  for  surface  normals. 
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6  Experimental  results:  laboratory  setup 

In  the  previous  sections,  we  described  the  three  algorithms  which  are  essential  for  analyzing  real 
color  images  taken  under  the  sun.  They  include  1.  the  separation  of  the  reflection  components 
from  the  two  light  sources:  the  sunlight  and  the  skylight,  2.  the  normalization  of  the  surface 
albedo  and  3.  the  unique  solution  for  surface  normals.  In  this  section,  we  applied  the  algorithms 
for  color  image  sequences  taken  in  a  laboratory  setup,  in  order  to  demonstrate  the  feasibility  of 
the  algorithms.  A  SONY  CCD  color  video  camera  module  model  XC-711  is  used  to  take  all 
images.  In  our  experimental  setup,  the  object  is  placed  at  the  center  of  the  origin  of  the  world 
coordinate  system,  and  the  color  video  camera  is  placed  at  the  top  of  the  object.  The  sunlight  is 
simulated  by  a  small  halogen  lamp  attached  to  a  PUMA  560  manipulator  which  moves  around  the 
object  on  its  equatorial  plane.  The  skylight  is  not  simulated  in  our  experimental  setup.  The  effect 
of  the  skylight  and  separation  of  the  reflection  components  from  the  skylight  will  be  described  in 
the  section  7. 

6.1  The  specular  reflection  color  estimation 

The  algorithms  to  estimate  the  illumination  color,  and  consequently,  to  separate  multiple  reflec¬ 
tion  components  are  applied  to  a  real  color  image  sequence.  A  shiny  ceramic  brooch  which  is 
painted  in  several  colors  is  used  in  this  experiment.  First,  a  sequence  of  color  images  was  taken  as 
the  point  light  source  was  moved  around  the  object  from  0^  =  -50°  to  0?  =  50  by  the  step  of 
10° .  As  described  in  the  section  3.2.1,  three  pixels  of  different  colors  in  the  image  are  manually 
selected.  The  6th  frame  of  the  color  image  sequence  and  the  manually  selected  pixels  are  shown 
in  Color  Figure  1.  The  observed  reflection  color  of  each  pixel  is  plotted  in  the  x-y  chromaticity 
diagram  over  the  image  sequence  to  obtain  a  line  segment  in  the  x-y  chromaticity  diagram  (Figure 
6).  Then,  the  illuminant  color  which  is  equal  to  the  specular  reflection  color  KsunS  can  then  be 

determined  by  the  intersection  of  the  three  line  segments.  In  this  experiment,  the  specular  reflec- 

T 

tion  color  vector  was  estimated  as  KsunS  =  (0.353,  0.334,  0.313)  . 
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Figure  6  Estimation  of  illuminant  color  in  the  x-y  chromaticity  diagram 
Those  three  pixels  of  different  colors  are  manually  selected  in  the  image  (Color  Figure  1). 

6.2  The  specular  reflection  component  removal 

The  algorithm  for  separating  the  specular  reflection  component  and  the  diffuse  reflection  compo¬ 
nent  from  the  sunlight,  described  in  the  section  3.2,  is  applied  to  the  input  color  image  sequence. 
As  stated  in  the  section  3.2,  the  algorithm  is  applied  at  each  pixel  basis.  For  each  pixel  in  the 
image,  the  red,  green  and  blue  intensities  over  the  image  sequence  are  initially  stored  in  the  matrix 
M  as  its  columns.  With  the  estimated  specular  reflection  color  vector  K  s  from  the  previous 
section,  the  diffuse  reflection  color  vector  KsunD  for  the  pixel  is  estimated  by  the  algorithm 
explained  in  the  section  3.2.2.  Then,  by  using  the  estimations  of  the  two  reflection  color  vectors, 
the  geometry  matrix  G  is  obtained  from  (EQ17).  Finally,  the  two  reflection  components  which 
are  observed  at  the  pixel  over  the  image  sequence  are  separated  by  using  (EQ19)  and  (EQ20). 

By  using  the  pixel-based  separation  algorithm,  we  can  generate  two  images  of  the  two  reflection 
components.  The  algorithm  is  applied  to  all  pixels  of  the  input  color  image  separately,  and  each 
separated  reflection  component  was  used  to  generate  the  diffuse  reflection  component  image 
sequence  and  the  specular  reflection  component  image  sequence.  The  6th  frames  of  the  diffuse 
reflection  component  image  sequence  and  the  specular  reflection  component  image  sequence  are 
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shown  in  Color  Figure  2  and  Color  Figure  3,  respectively. 

6.3  Albedo  normalization 

After  the  specular  reflection  component  from  the  sunlight  is  removed  from  the  color  image 
sequence,  we  obtain  the  color  image  sequence  which  includes  only  the  diffuse  reflection  compo¬ 
nent  from  the  sunlight.  The  algorithm  to  normalize  the  surface  albedo  on  the  object  surface, 
described  in  the  section  4  is  applied  to  the  resulting  image  sequence.  Note  that  only  one  color 
band  (for  examples,  in  our  case,  the  red  band)  is  used  for  the  normalization  of  the  surface  albedo 
and  the  shape  recovery.  This  is  because  intensity  profiles  of  all  of  the  three  color  bands  are  equiv¬ 
alent  in  terms  of  shape  recovery  when  the  images  include  only  the  diffuse  reflection  component. 
Therefore,  those  three  color  bands  are  redundant,  and  using  only  one  band  is  sufficient. 

The  6th  frame  of  the  diffuse  reflection  image  sequence  which  is  shown  in  Color  Figure  2  is  used 
here.  First,  by  detecting  boundary  edges,  the  object  surface  is  segmented  into  subregions,  each  of 
which  has  a  uniform  surface  albedo.  The  boundary  edges  are  defined  as  a  group  of  pixels  where 
pixel  intensities  change  abruptly.  On  the  other  hand,  pixel  intensities  change  gradually  within 
each  of  the  subregions.  After  the  segmentation  of  the  object  surface,  all  regions  are  uniquely 
labeled  by  using  the  sequential  labeling  algorithm.  The  result  of  the  region  segmentation  and  the 
sequential  labeling  is  shown  in  Figure  7. 


Figure  7  Segmented  uniform  albedo  regions 
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Figure  8  The  frame  6  with  normalized  albedo  in  red  band 

Then,  surface  albedo  ratios  between  all  pairs  of  two  neighboring  subregions  are  computed.  Con¬ 
sider  that  we  compute  the  surface  albedo  ratio  between  the  region  A  and  the  region  B .  For  each 
pixel  of  the  region  A  which  exists  at  the  boundary  between  the  two  regions,  the  pixel  of  the 
region  B  which  is  closest  to  the  pixel  in  the  region  A  is  selected.  Then,  the  ratio  of  the  two  pixel 
intensities  is  used  as  the  surface  reflectance  ratio  (EQ25)  between  the  region  A  and  the  region  B . 
In  order  to  obtain  a  more  accurate  estimation  of  the  ratio,  all  pixels  along  the  boundary  are  used  to 
compute  the  average  of  the  ratio.  The  average  ratio  is  used  as  the  surface  ratio  between  the  region 
A  and  the  region  B .  The  same  procedure  is  repeated  for  all  pairs  of  two  neighboring  regions  to 
build  a  table  of  the  surface  albedo  ratio.  Then,  the  surface  albedo  ratios  are  propagated  in  the 
table,  so  that  the  surface  albedo  ratios  of  all  pairs  of  any  two  non-isolated  regions  are  obtained. 
Finally,  the  computed  surface  albedo  ratios  are  used  for  normalizing  image  intensities  of  all 
frames  in  the  image  sequence.  Figure  8  shows  the  normalization  result.  All  pixel  intensities  of  the 
object  surface  are  normalized  as  if  the  object  had  a  uniform  surface  albedo  corresponding  to  the 
region  1  in  Figure  7.  This  normalized  diffuse  reflection  image  is  used  for  recovering  the  object 
shape. 

6.4  Unique  solution  for  surface  normal 

In  this  section,  the  algorithm  to  determine  surface  normals  uniquely,  which  was  described  in  the 
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section  5,  is  applied  to  a  sequence  of  real  color  images  taken  in  our  laboratory  setup.  A  plastic 
dinosaur  face  is  used  for  this  experiment.  The  sequence  of  color  images  was  taken  as  the  point 
light  source  was  moved  around  the  object  from  0^,  =  -70°  to  0^  =  70°  by  the  step  of  5° .  The 
specular  reflection  component  is  removed  from  the  input  image  sequence  by  using  the  same  algo¬ 
rithm  used  in  the  section  6.2.  In  this  experiment,  the  specular  reflection  color  was  directly  mea¬ 
sured  rather  than  estimating  as  described  in  the  section  6.1.  The  8th  frame  of  the  resulting  diffuse 
reflection  image  sequence  is  shown  in  Color  Figure  4. 

The  algorithm  for  obtaining  two  sets  of  surface  normals  which  was  described  in  the  section  5.1 
was  applied  to  the  red  band  of  the  resulting  diffuse  reflection  image  sequence.  Computed  two  sets 
of  surface  normals  n  ]  and  «2  are  shown  in  Figure  9  as  a  needle  diagram.  Subsequently,  the  inte- 
grability  constraint  was  applied  to  determine  the  correct  set  of  surface  normals  uniquely.  First,  the 
object  surface  was  segmented  into  subregions  by  defining  a  boundary  where  the  two  surface  nor¬ 
mals  and  are  similar.  The  obtained  boundary  is  shown  in  Figure  10.  Theoretically,  the 
boundary  should  be  connected  and  narrow.  However,  in  a  practical  case,  the  obtained  boundary 
tends  to  be  wide  in  order  to  guarantee  its  connectivity.  Thus,  the  thinning  operation,  in  our  case 
the  medial  axis  transformation,  was  applied  to  narrow  the  boundary.  Figure  1 1  shows  the  resulting 
boundary  after  the  medial  axis  transformation. 


Figure  9  Two  sets  of  surface  normals 
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Figure  10  The  boundary  region  obtained  from  two  surface  normal  sets 


Figure  11  The  boundary  after  medial  axis  transformation 

Then,  a  standard  sequential  labeling  algorithm  was  applied  to  the  segmented  regions  to  assign 
unique  labels  as  illustrated  in  Figure  12.  In  the  figure,  the  labels  are  represented  by  different  gray 
levels.  Finally,  the  integrability  constraint  is  applied  to  obtain  the  correct  set  of  surface  normals  in 
each  labeled  region  as  explained  in  the  section  5.2.  The  unique  surface  normal  solution  is  shown 
in  Figure  13.  Finally,  the  object  surface  shape  was  recovered  by  using  a  standard  height  from  nor¬ 
mals  procedure.  In  our  implementation,  the  relaxation  method  proposed  by  Horn  and  Brooks  [5] 
was  used.  Figure  14  shows  the  surface  shape  of  the  dinosaur’s  face. 
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Figure  12  Segmented  regions  (gray  levels  represent  regions) 


Figure  13  Right  surface  normal  set 


Figure  14  Recovered  object  shape 


) 
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7  Experimental  result:  outdoor  scene  (water  tower) 

In  this  section,  we  demonstrate  successful  reflection  component  separation  and  shape  recovery 
under  solar  illumination.  To  this  end,  the  algorithms  described  in  this  paper  were  applied  to  real 
images  of  a  water  tower  (Color  Figure  5)  taken  in  an  outdoor  environment  under  the  sun.  A  clear 
day  was  chosen  for  taking  the  images  in  Pittsburgh,  PA  to  avoid  the  undesirable  effects  of  clouds 
in  the  sky.  A  SONY  CCD  color  video  camera  modules  model  XC-711  was  used  to  take  all 
images,  and  the  images  are  recorded  by  using  a  U-matic  video  recorder.  Later,  the  images  are  dig¬ 
itized  from  the  recorded  U-matic  tape  every  fifteen  minutes.  The  first  image  frame  was  taken  at 
10:45am,  and  the  last  frame  was  taken  at  4:15pm.  In  this  manner,  we  took  23  frames  of  color 
images  in  total.  As  an  example,  the  image  frame  taken  at  2:15pm  is  shown  in  Color  Figure  5. 

First,  the  region  of  interest  corresponding  to  the  water  tower  was  extracted.  The  skylight  is  almost 
uniform  through  a  day  from  the  sunrise  until  the  sunset.  As  a  result,  all  background  pixels  which 
corresponds  to  the  sky  can  be  defined  as  pixels  with  low  intensity  variation.  At  each  pixel  in  the 
image,  the  variance  of  the  pixel  intensity  over  the  time  sequence  is  computed.  Then,  all  pixels 
with  little  intensity  variation  are  removed  by  applying  a  simple  threshold,  in  order  to  extract  the 
region  of  interest  for  the  further  processing.  The  extracted  region  of  interest  is  shown  in  Color 
Figure  6. 

The  next  step  is  to  remove  the  reflection  component  from  the  skylight.  According  to  the  reflection 
model  under  the  solar  illumination  (EQ12)  that  we  developed  in  the  section  2.2,  the  two  reflection 
components  due  to  the  skylight  are  represented  as  a  constant  value  Ksky .  The  constant  value 
Ksky  can  be  estimated  as  an  average  pixel  color  of  a  uniform  intensity  region  which  is  in  a 
shadow  from  the  sunlight.  In  our  experiment,  the  region  of  a  constant  pixel  colors  is  selected  man¬ 
ually  as  shown  in  Color  Figure  6.  The  measured  pixel  color  within  the  region  is 
(r,g,b)  =  (14.8,  17.2,  19.5)  with  the  variance  (0.2,  0.3,  0.6)  .  The  pixel  color  vector  was 
subtracted  from  intensities  of  all  pixels  to  eliminate  effects  from  the  skylight.  After  this  operation, 
the  color  images  have  only  the  reflection  components  due  to  the  sunlight.  The  resulting  image  is 
shown  in  Color  Figure  7.  It  can  be  seen  that  the  image  has  more  contrast  between  an  illuminated 
region  and  a  shadow  region,  compared  to  the  image  with  the  reflection  component  due  to  the  sky¬ 
light  (Color  Figure  6).  All  of  frames  of  the  input  color  images  are  processed  in  the  same  manner 
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to  remove  the  reflection  component  due  to  the  skylight. 

After  the  removal  of  the  reflection  component  from  the  skylight,  the  sequence  of  color  images 
includes  two  reflection  component:  the  diffuse  reflection  component  and  the  specular  reflection 
component  due  to  the  sunlight  as  modeled  by  (EQ13).  The  algorithm  to  remove  the  specular 
reflection  component  explained  in  the  section  3.2  was  applied  to  the  resulting  color  images.  At 
each  pixel  in  the  color  image,  the  two  reflection  components  were  separated  and  only  the  diffuse 
reflection  component  was  used  for  further  shape  recovery.  As  an  example,  one  frame  of  the  result¬ 
ing  color  image  sequence  is  shown  in  Color  Figure  8.  The  image  includes  only  one  reflection 
component:  the  diffuse  reflection  component  from  the  sunlight.  The  water  tower  appears  to  have  a 
uniform  surface  albedo.  Therefore,  it  was  not  necessary  to  apply  the  surface  albedo  normalization 
procedure  explained  in  the  section  4  in  this  experiment. 

The  algorithm  to  determine  surface  normals  uniquely  by  using  an  image  sequence  was  applied  to 
the  red  band  of  the  resulting  color  image  sequence.  Figure  15  shows  the  recovered  surface  nor¬ 
mals  of  the  water  tower.  Note  that  surface  normals  are  not  obtained  in  the  lower  right  part  of  the 
water  tower.  This  is  because,  in  the  region,  the  maximum  intensity  is  not  observed  at  each  pixel 
through  the  image  sequence.  To  recover  surface  normals  in  the  region,  we  need  to  take  an  input 
image  sequence  over  a  longer  period  of  time  than  this  experiment.  Also,  another  techniques  such 
as  photometric  stereo  can  be  used  for  recovering  surface  normals  in  the  region.  However,  in  this 
case,  it  will  be  hard  to  estimate  surface  normals  accurately.  Finally,  the  relaxation  method  for  cal¬ 
culating  height  from  surface  normals  was  applied.  The  recovered  shape  of  the  part  of  the  water 
tower  is  shown  in  Figure  16. 


Figure  15  Surface  normals 


Figure  16  Recovered  shape  of  the  water  tank 
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8  Conclusions 

We  studied  image  analysis  of  an  outdoor  scene  in  this  paper.  The  scope  of  traditional  techniques 
for  reflectance  analysis  such  as  shape  from  shading  and  photometric  stereo  has  been  highly 
restricted  to  images  taken  under  a  controlled  laboratory  setup.  It  is  attributed  to  the  fundamental 
difficulties  in  analyzing  real  images  taken  in  an  outdoor  environment.  First,  we  addressed  the  dif¬ 
ficulties  involved  in  analyzing  real  outdoor  images  under  the  solar  illumination.  They  include  1. 
multiple  reflection  components  due  to  multiple  light  sources  of  different  spectral  and  spatial  dis¬ 
tribution,  namely  the  sunlight  and  the  skylight,  2.  non-uniform  surface  albedos  and  3.  ambiguity 
in  surface  normal  determination  caused  by  the  sun’s  restricted  motion.  For  each  of  those  three 
problems,  solutions  were  proposed  based  on  the  reflectance  model  under  the  solar  illumination 
which  we  developed,  Finally,  the  effectiveness  of  the  algorithms  were  successfully  demonstrated 
by  using  real  color  images  taken  both  in  a  laboratory  setup  simulating  the  sunlight  and  in  an  out¬ 
door  environment.  We  believe  this  is  one  of  the  first  attempts  for  analyzing  reflection  on  object 
surfaces  in  an  outdoor  scene. 
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Color  figures 


Color  Figure  1  Input  color  image  sequence  (frame  Color  Figure  3  specular  reflection  component  of 
6  is  shown)  and  manually  selected  pixels  the  frame  6 


Color  Figure  2  Diffuse  reflection  component  of  the  Color  Figure  4  Diffuse  reflection  component  image 

frame  6  (frame  8) 


Color  Figure  5  Outdoor  intensity  image  (water 
tank  frame  15) 


Region  of  constant  color 


\ 


Color  Figure  6  Extracted  region  of  interest 


Color  Figure  7  Water  tank  image  without  sky 
reflection  component 


Color  Figure  8  Water  tank  image  after  highlight 
removal 
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